{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning Tutorial with Keras and Tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div>\n",
    "    <img style=\"text-align: left\" src=\"imgs/keras-tensorflow-logo.jpg\" width=\"40%\" />\n",
    "<div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the Materials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"imgs/github.jpg\" />\n",
    "```shell\n",
    "\n",
    "git clone https://github.com/leriomaggio/deep-learning-keras-tensorflow.git\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Outline at a glance\n",
    "\n",
    "- **Part I**: **Introduction**\n",
    "\n",
    "    - Intro to Artificial Neural Networks\n",
    "        - Perceptron and MLP    \n",
    "        - naive pure-Python implementation\n",
    "        - fast forward, sgd, backprop\n",
    "        \n",
    "    - Introduction to Deep Learning Frameworks\n",
    "        - Intro to Theano\n",
    "        - Intro to Tensorflow\n",
    "        - Intro to Keras\n",
    "            - Overview and main features\n",
    "            - Overview of the `core` layers\n",
    "            - Multi-Layer Perceptron and Fully Connected\n",
    "                - Examples with `keras.models.Sequential` and `Dense`\n",
    "            - Keras Backend\n",
    "    \n",
    "- **Part II**: **Supervised Learning **\n",
    "    \n",
    "    - Fully Connected Networks and Embeddings\n",
    "        - Intro to MNIST Dataset\n",
    "        - Hidden Leayer Representation and Embeddings\n",
    "        \n",
    "    - Convolutional Neural Networks\n",
    "        - meaning of convolutional filters\n",
    "            - examples from ImageNet    \n",
    "        - Visualising ConvNets \n",
    "\n",
    "        - Advanced CNN\n",
    "            - Dropout\n",
    "            - MaxPooling\n",
    "            - Batch Normalisation\n",
    "\n",
    "        - HandsOn: MNIST Dataset\n",
    "            - FC and MNIST\n",
    "            - CNN and MNIST\n",
    "        \n",
    "        - Deep Convolutiona Neural Networks with Keras (ref: `keras.applications`)\n",
    "            - VGG16\n",
    "            - VGG19\n",
    "            - ResNet50\n",
    "    - Transfer Learning and FineTuning\n",
    "    - Hyperparameters Optimisation \n",
    "        \n",
    "- **Part III**: **Unsupervised Learning**\n",
    "\n",
    "    - AutoEncoders and Embeddings\n",
    "\t- AutoEncoders and MNIST\n",
    "    \t- word2vec and doc2vec (gensim) with `keras.datasets`\n",
    "        - word2vec and CNN\n",
    "    \n",
    "- **Part IV**: **Recurrent Neural Networks**\n",
    "    - Recurrent Neural Network in Keras \n",
    "        -  `SimpleRNN`, `LSTM`, `GRU`\n",
    "    - LSTM for Sentence Generation\n",
    "\t\t\n",
    "- **PartV**: **Additional Materials**:  \n",
    "   - Custom Layers in Keras \n",
    "   - Multi modal Network Topologies with Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial requires the following packages:\n",
    "\n",
    "- Python version 3.5\n",
    "    - Python 3.4 should be fine as well\n",
    "    - likely Python 2.7 would be also fine, but *who knows*? :P\n",
    "    \n",
    "- `numpy` version 1.10 or later: http://www.numpy.org/\n",
    "- `scipy` version 0.16 or later: http://www.scipy.org/\n",
    "- `matplotlib` version 1.4 or later: http://matplotlib.org/\n",
    "- `pandas` version 0.16 or later: http://pandas.pydata.org\n",
    "- `scikit-learn` version 0.15 or later: http://scikit-learn.org\n",
    "- `keras` version 2.0 or later: http://keras.io\n",
    "- `tensorflow` version 1.0 or later: https://www.tensorflow.org\n",
    "- `ipython`/`jupyter` version 4.0 or later, with notebook support\n",
    "\n",
    "(Optional but recommended):\n",
    "\n",
    "- `pyyaml`\n",
    "- `hdf5` and `h5py` (required if you use model saving/loading functions in keras)\n",
    "- **NVIDIA cuDNN** if you have NVIDIA GPUs on your machines.\n",
    "    [https://developer.nvidia.com/rdp/cudnn-download]()\n",
    "\n",
    "The easiest way to get (most) these is to use an all-in-one installer such as [Anaconda](http://www.continuum.io/downloads) from Continuum. These are available for multiple architectures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python Version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm currently running this tutorial with **Python 3** on **Anaconda**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python 3.5.2\r\n"
     ]
    }
   ],
   "source": [
    "!python --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Keras with tensorflow\n",
    "\n",
    "1) Create the `keras.json` (if it does not exist):\n",
    "\n",
    "```shell\n",
    "touch $HOME/.keras/keras.json\n",
    "```\n",
    "\n",
    "2) Copy the following content into the file:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"epsilon\": 1e-07,\n",
    "    \"backend\": \"tensorflow\",\n",
    "    \"floatx\": \"float32\",\n",
    "    \"image_data_format\": \"channels_last\"\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "\t\"epsilon\": 1e-07,\r\n",
      "\t\"backend\": \"tensorflow\",\r\n",
      "\t\"floatx\": \"float32\",\r\n",
      "\t\"image_data_format\": \"channels_last\"\r\n",
      "}"
     ]
    }
   ],
   "source": [
    "!cat ~/.keras/keras.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test if everything is up&running"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Check import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy as sp\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Check installeded Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numpy: 1.11.1\n",
      "scipy: 0.18.0\n",
      "matplotlib: 1.5.2\n",
      "iPython: 5.1.0\n",
      "scikit-learn: 0.18\n"
     ]
    }
   ],
   "source": [
    "import numpy\n",
    "print('numpy:', numpy.__version__)\n",
    "\n",
    "import scipy\n",
    "print('scipy:', scipy.__version__)\n",
    "\n",
    "import matplotlib\n",
    "print('matplotlib:', matplotlib.__version__)\n",
    "\n",
    "import IPython\n",
    "print('iPython:', IPython.__version__)\n",
    "\n",
    "import sklearn\n",
    "print('scikit-learn:', sklearn.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "keras:  2.0.2\n",
      "Theano:  0.9.0\n",
      "Tensorflow:  1.0.1\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "print('keras: ', keras.__version__)\n",
    "\n",
    "# optional\n",
    "import theano\n",
    "print('Theano: ', theano.__version__)\n",
    "\n",
    "import tensorflow as tf\n",
    "print('Tensorflow: ', tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<h1 style=\"text-align: center;\">If everything worked till down here, you're ready to start!</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
